Papers with knowledge distillation techniques
CocaCLIP: Exploring Distillation of Fully-Connected Knowledge Interaction Graph for Lightweight Text-Image Retrieval (2023.acl-industry)
Copied to clipboard
| Challenge: | Existing methods for text-image retrieval are limited to edge devices and real-time situations due to the substantial indexing and inference time. |
| Approach: | They propose a fully-Connected knowledge interaction graph technique for cross-modal pre-training distillation. |
| Outcome: | The proposed method achieves SOTA performances on the widely-used Flickr30K and MSCOCO benchmarks under the lightweight setting. |
Thinking with DistilQwen: A Tale of Four Distilled Reasoning and Reward Model Series (2025.emnlp-industry)
Copied to clipboard
| Challenge: | In the rapidly evolving landscape of large language models, the need for efficient reasoning models has become increasingly urgent. |
| Approach: | They extend the Qwen model family by introducing four model series specifically designed for industrial applications. |
| Outcome: | The proposed models outperform previous models in multiple benchmarks and provide scalable training and inference functionality on the Alibaba Cloud PAI platform. |
TransferCVLM: Transferring Cross-Modal Knowledge for Vision-Language Modeling (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Recent large vision-language multimodal models pre-trained with huge amount of image-text pairs show remarkable performances in downstream tasks. |
| Approach: | They propose a method of efficient knowledge transfer that integrates pre-trained uni-modal models into a combined vision-language model without pre-training . they propose to fine-tune the model and transfer multimodal knowledge from a teacher vision-linguistic model to the CVLM for each task application. |
| Outcome: | The proposed method outperforms existing vision-language models in downstream tasks. |
PairDistill: Pairwise Relevance Distillation for Dense Retrieval (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent advances in dense retrieval have demonstrated remarkable efficacy compared to traditional sparse retrieval methods. |
| Approach: | They propose to use pairwise relevance distillation to leverage pairwise reranking to enrich the training of dense retrieval models. |
| Outcome: | The proposed method outperforms existing methods and achieves state-of-the-art results on multiple benchmarks. |
Comprehensive and Efficient Distillation for Lightweight Sentiment Analysis Models (2025.emnlp-main)
Copied to clipboard
| Challenge: | Recent efforts to develop lightweight and practical sentiment analysis models are limited by manual instruction and large-scale user texts. |
| Approach: | They propose a framework for sentiment analysis that uses attribute-based instruction construction and difficulty-based data filtering to distill knowledge. |
| Outcome: | The proposed framework outperforms baseline methods in data efficiency and performance. |